Image processing apparatus for detecting moving subject, image processing method, and storage medium

ABSTRACT

An image processing apparatus configured to detect a moving subject region includes a likelihood generation unit, a similarity detection unit, and a correction unit. The likelihood generation unit detects a motion of a region in an image and, based on at least two input images, generates a moving subject likelihood for each region. The similarity detection unit detects a similarity between a target region and a peripheral region of the target region for at least one of the input images. The correction unit corrects the moving subject likelihood of the target region based on the detected similarity and the generated moving subject likelihood of the peripheral region. The moving subject region is detected based on the corrected moving subject likelihood.

BACKGROUND Field

The present disclosure relates to an image processing technique fordetecting a moving subject region from an image.

Description of the Related Art

In recent years, digital cameras, digital camcorders, and many otherimaging apparatuses for combining a plurality of images and recording acombined image have been commercially produced. Some of these imagingapparatuses have a function of generating a combined image with reducedrandom noise by combining a plurality of images captured at differenttimes. This allows a user to obtain a combined image with random noisefurther reduced than an uncombined image. However, if a subject moveswhen capturing a plurality of images with such a combination functionset to ON, a combined image generated may contain multiple images of themoving subject. As a technique for restraining the generation of suchmultiple images, a certain technique is known to inhibit combinationprocessing in a region where the movement of a subject is detected.

As a technique for detecting a moving subject region such as a movingsubject, for example, Japanese Patent Application Laid-Open No.2013-62741 discusses a technique for detecting a moving subject regionbased on the difference absolute value between a plurality of imagesreduced with a reduction ratio determined according to the amount ofcamera shake. The technique discussed in Japanese Patent ApplicationLaid-Open No. 2013-62741 reduces not only positional deviations ofstationary subjects due to camera shake by changing the image reductionratio based on the amount of camera shake but also the influence ofnoise by reducing an image, thus improving the detection accuracy of amoving subject region. Japanese Patent Application Laid-Open No.2011-discusses a technique for selecting low-resolution images (i.e.,reduced images) according to positional deviations remaining afterpositioning a plurality of images, and detecting a moving subject regionbased on the difference absolute value between the plurality of selectedlow-resolution images. In the technique discussed in Japanese PatentApplication Laid-Open No. 2011-198241, low-resolution images areselected according to positional deviations remaining after positioninga plurality of images to reduce positional deviations of stationarysubjects due to camera shake, thus improving the detection accuracy of amoving subject region.

However, with an image containing much random noise, such as an imagecaptured with high sensitivity, the difference absolute value betweenimages is increased by the influence of random noise. In this case, itmay be difficult to distinguish between a moving subject region andrandom noise. In particular, in a case where the difference absolutevalue between images is smaller than the difference absolute valuecaused by random noise, it may be difficult to detect a moving subjectpossibly resulting in incorrect detection of a moving subject region.When detecting a moving subject region based on reduced images as inJapanese Patent Application Laid-Open No. 2013-62741 and Japanese PatentApplication Laid-Open No. 2011-198241 described above, the detectedmoving subject region will be subsequently enlarged into the originalfull-size image. Therefore, if a moving subject region is incorrectlydetected, for example, a stationary region around the moving subject maybe incorrectly handled as a moving subject region or, conversely, a partof a moving subject region may be incorrectly handled as a stationaryregion.

SUMMARY

The present disclosure is directed to improving the detection accuracyof a moving subject region such as a moving subject.

According to an aspect of the present disclosure, an image processingapparatus configured to detect a moving subject region includes a memorythat stores instructions, and one or more processors configured toexecute the instructions to cause the image processing apparatus tofunction as: a likelihood generation unit configured to detect a motionof a region in an image and, based on at least two input images,generate a moving subject likelihood for each region, a similaritydetection unit configured to detect a similarity between a target regionand a peripheral region of the target region for at least one of theinput images, and a correction unit configured to correct the movingsubject likelihood of the target region based on the detected similarityand the generated moving subject likelihood of the peripheral region,wherein the moving subject region is detected based on the correctedmoving subject likelihood.

Further features of the present disclosure will become apparent from thefollowing description of embodiments with reference to the attacheddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an overall configuration of an imaging apparatusaccording to embodiments.

FIG. 2 illustrates an example configuration of a combined imagegeneration unit.

FIG. 3 is a flowchart illustrating a processing flow of the combinedimage generation unit.

FIGS. 4A, 4B, and 4C illustrate examples of a standard image, areference image, and a positioned reference image, respectively.

FIG. 5 illustrates a combination ratio curve of the standard image.

FIG. 6 illustrates an example configuration of a moving subject regiondetection unit according to a first embodiment.

FIG. 7 is a flowchart illustrating processing of the moving subjectregion detection unit according to the first embodiment.

FIG. 8 illustrates a moving subject likelihood curve.

FIGS. 9A and 9B illustrate similarity weight coefficient calculationprocessing.

FIG. 10 illustrates moving subject likelihood weight coefficientacquisition processing.

FIGS. 11A and 11B illustrate integrated weight coefficient calculationprocessing.

FIG. 12 illustrates an example configuration of a moving subject regiondetection unit according to a second embodiment.

FIG. 13 is a flowchart illustrating processing of the moving subjectregion detection unit according to the second embodiment.

FIG. 14 illustrates full-size image data.

FIGS. 15A to 15H illustrate hierarchically corrected moving subjectlikelihood generation processing.

FIG. 16 illustrates an edge degree curve.

FIG. 17 illustrates a combination ratio curve of a hierarchicallycorrected moving subject likelihood.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present disclosure will be described below withreference to the accompanying drawings.

The image processing apparatus according to embodiments of the presentdisclosure are applicable, for example, to an imaging apparatus having acombination function of combining a plurality of images captured atdifferent time points. The combination function combines a plurality ofimages captured at different times to generate a combined image andrecords the combined image.

FIG. 1 illustrates an overall configuration of an imaging apparatus 100as an example application of the image processing apparatus according tothe present embodiment. In the imaging apparatus 100 illustrated in FIG.1, for example, the image processing apparatus according to the presentembodiment is equivalent to an image processing unit 107. For example,when the combination function is set to ON, the image processing unit107 performs image combination processing based on the moving subjectlikelihood and the corrected moving subject likelihood (describedbelow).

Referring to FIG. 1, the control unit 101 is, for example, a centralprocessing unit (CPU) which reads an operation program for eachcomponent of the imaging apparatus 100 from a read only memory (ROM)102, loads the program into a random access memory (RAM) 103, andexecutes the program to control the operation of each component of theimaging apparatus 100. The ROM 102 is electrically erasable andrecordable nonvolatile memory for storing the operation program for eachcomponent of the imaging apparatus 100 and also storing parametersrequired for the operation of each component and processing for variouscalculations (described below). The RAM 103 is rewritable volatilememory used as a temporary storage area for data output during operationof each component of the imaging apparatus 100. The combination ratiocurve, moving subject likelihood curve, similarity weight coefficientcurve, moving subject likelihood weight coefficient curve, integratedweight coefficient curve, edge degree curve, etc. may be prestored asparameters in the ROM 102, and may be generated by program execution.

An optical system 104 formed by a lens group including a zoom lens and afocal lens forms a subject image on an imaging unit 105 (describedbelow). The imaging unit 105 includes an image sensor such as a chargecoupled device (CCD) sensor and a complementary metal oxidesemiconductor (CMOS) sensor, and color filters. The imaging unit 105performs photoelectric conversion of an optical image formed on theimaging unit 105 by the optical system 104 and outputs an acquiredanalog image signal to an analog-to-digital (A/D) converter 106. The A/Dconverter 106 converts the input analog image signal into a digitalimage signal and outputs obtained digital image data to the RAM 103. TheA/D converter 106 includes an amplifier for amplifying an analog imagesignal or a digital image signal based on the amplification factor(sensitivity information) determined by the control unit 101.

The image processing unit 107 applies various image processing such aswhite balance adjustment, color interpolation, and gamma processing onthe image data stored in the RAM 103. According to the presentembodiment, the image processing unit 107 includes a combined imagegeneration unit 200 (described below). The combined image generationunit 200 acquires a standard image and a reference image as input imagesfrom a plurality of images in the RAM 103 and performs positioning ofthe reference image based on the standard image. Then, based on thepositioned reference image and the standard image, the combined imagegeneration unit 200 detects a moving region such as a subject moving inthe images. The combined image generation unit 200 further calculatesthe moving subject likelihood based on the standard image and thepositioned reference image and, based on a corrected moving subjectlikelihood which is obtained by correcting the moving subjectlikelihood, combines the standard image and the positioned referenceimage to generate a combined image.

A recording unit 108 is, for example, a detachably attached memory cardfor recording images processed by the image processing unit 107 asrecorded images through the RAM 103.

A display unit 109 is a display device such as a liquid crystal display(LCD) which displays images recorded in the RAM 103 and the recordingunit 108 and displays an operation user interface for receiving userinstructions.

Although not illustrated in FIG. 1, the imaging apparatus 100 isprovided with various operation buttons such as a power switch, ashutter button, a menu button, and a playback button provided on commoncameras.

Detail operations of the image processing unit 107 included in theimaging apparatus 100 according to the present embodiment will bedescribed below. The present embodiment will be described belowcentering on a case where the image processing unit 107 combines twodifferent images based on the corrected moving subject likelihood(described below). More specifically, the image processing unit 107performs image processing for generating a combined image with reducedrandom noise while preventing the generation of multiple images of amoving subject.

The image processing unit 107 includes the combined image generationunit 200 as illustrated in FIG. 2.

The combined image generation unit 200 is a component included in thecombination function of combining data of two images stored in the RAM103, to generate a combined image. The combined image generation unit200 includes a positioning unit 201, a moving subject region detectionunit 202, and an image combination unit 203, as illustrated in FIG. 2.Image data and various information indicating values such as the movingsubject likelihood (described below) are transmitted between theabove-described units illustrated in FIG. 2. Hereinafter, descriptionsof these pieces of data and information will be omitted forsimplification. This also applies to descriptions of drawings indicatingother configurations (described below).

Processing performed by the imaging apparatus 100 illustrated in FIG. 1and the combined image generation unit 200 (in the image processing unit107) illustrated in FIG. 2 will be described below with reference to theflowchart illustrated in FIG. 3. This also applies to other flowcharts(described below).

Referring to FIG. 3, in step S301, the control unit 101 of the imagingapparatus 100 illustrated in FIG. 1 selects and acquires a standardimage and a reference image to be combined with the standard image, outof a plurality of images stored in the RAM 103. For example, the controlunit 101 acquires as a standard image the first image capturedimmediately after pressing the shutter button and acquires the secondand subsequent images as reference images during image capturing. Then,the control unit 101 transmits the standard image and the referenceimage acquired from the RAM 103 in step S301, to the image processingunit 107.

In step S302, the positioning unit 201 performs positioning processingfor aligning the position of the reference image with the position ofthe standard image. More specifically, the positioning unit 201 detectsa moving vector between the standard image and the reference imageacquired in step S301 and performs geometric deformation on thereference image based on the motion vector.

The positioning processing will be described below with reference toFIGS. 4A to 4C. FIG. 4A illustrates a standard image 400, FIG. 4Billustrates a reference image 401, and FIG. 4C illustrates a positionedreference image 402 having undergone the positioning processing.

The reference image 401 illustrated in FIG. 4B is an image captured atdifferent times from the standard image 400 illustrated in FIG. 4A. Forexample, in the reference image 401, positions and inclinations aredeviated with respect those in the standard image 400 illustrated inFIG. 4A due to camera shake during image capturing. The positioning unit201 corrects such deviations of positions and inclinations through thepositioning processing. The positioning unit 201 first detects a motionvector indicating an overall motion between the standard image 400illustrated in FIG. 4A and the reference image 401 illustrated in FIG.4B. Examples of motion vector detection methods include the blockmatching method. Then, the positioning unit 201 calculates a geometricconversion factor A represented by formula (1) as a coefficient forperforming geometric conversion on the reference image 401 based on thedetected motion vector.

$\begin{matrix}{A = \begin{pmatrix}a & b & c \\d & e & f \\g & h & i\end{pmatrix}} & {{Formula}\mspace{14mu}(1)}\end{matrix}$

By using the geometric conversion factor A, the positioning unit 201performs geometric deformation calculations represented by formula (2)for the reference image 401 illustrated in FIG. 4B to generate thepositioned reference image 402 illustrated in FIG. 4C. Referring toformula (2), the reference image 401 is denoted by (x-coordinate,y-coordinate), and the positioned reference image 402 is denoted by I′(x′-coordinate, y′-coordinate).

$\begin{matrix}{I^{\prime} = {\begin{pmatrix}x^{\prime} \\y^{\prime} \\1\end{pmatrix} = {{AI} = {\begin{pmatrix}a & b & c \\d & e & f \\g & h & i\end{pmatrix}\begin{pmatrix}x \\y \\1\end{pmatrix}}}}} & {{Formula}\mspace{14mu}(2)}\end{matrix}$

The positioning unit 201 performs such processing for adjustingpositions and inclinations to allow adjusting the positions andinclinations of stationary subjects (for example, buildings and trees)between the standard image 400 and the reference image 401, as in thepositioned reference image 402 illustrated in FIG. 4C.

Referring back to the flowchart in FIG. 3, in step S303, the movingsubject region detection unit 202 compares the standard image 400 andthe positioned reference image 402 to detect a moving region in theimages. The moving subject region detection unit 202 further obtains themoving subject likelihood for each of predetermined regions (forexample, for each pixel) in the detected moving region based on thestandard image 400 and the positioned reference image 402. According tothe present embodiment, a moving region is represented by multi-valuedata representing the likelihood of being a moving subject (i.e., movingsubject likelihood). According to the present embodiment, in thedetected moving region and a region neighboring the moving region, apixel having a moving subject likelihood value of 100 or more isregarded as a pixel of a moving subject region, and a pixel having amoving subject likelihood value of 0 is regarded as a pixel of astationary region which is not a moving subject. The larger the movingsubject likelihood value of a pixel is with respect to 0, the more thepixel is likely to be included in a moving subject region. The movingsubject region detection unit 202 according to the present embodimentcorrects the detected moving subject likelihood (described below) togenerate the corrected moving subject likelihood. The detailedconfiguration, the moving subject likelihood generation processing, andthe corrected moving subject likelihood generation processing of themoving subject region detection unit 202 will be described below.

In step S304, as represented by the following formula (3), the imagecombination unit 203 sets the combination ratio based on the correctedmoving subject likelihood and, based on the combination ratio, combinesthe standard image 400 with the positioned reference image 402 for eachpixel to generate a combined image.P=w*Pbase+(1−w)*Pref  Formula (3)

Referring to formula (3), Pbase denotes the pixel value of the standardimage 400, Pref denotes the pixel value of the positioned referenceimage 402, w denotes the combination ratio of the standard image 400,and P denotes the pixel value of the combined image.

The combination ratio setting processing based on the corrected movingsubject likelihood will be described below with reference to FIG. 5.FIG. 5 illustrates a combination ratio curve representing the relationbetween the corrected moving subject likelihood and the combinationratio of the standard image 400. Referring to FIG. 5, the vertical axisindicates the combination ratio of the standard image 400, and thehorizontal axis indicates the corrected moving subject likelihood. Thecombination ratio curve illustrated in FIG. 5 is set so that thecombination ratio of the standard image 400 increases with increasingcorrected moving subject likelihood.

According to the combination ratio curve illustrated in FIG. 5, in amoving subject region where the value of the corrected moving subjectlikelihood is 100 or more, the combination ratio of the standard image400 is set to 100%. Reference image combination processing is inhibitedfor that moving subject region. This restricts the generation ofmultiple images. On the other hand, in a stationary region where thevalue of the corrected moving subject likelihood is 0, the combinationratio of the standard image 400 is set to 50%. The stationary region ofthe reference image 401 is combined with the standard image 400 with acombination ratio of 50%. In this case, the combination of the referenceimage 401 reduces random noise.

The detailed configuration, the moving subject likelihood generationprocessing, and the corrected moving subject likelihood generationprocessing of the moving subject region detection unit 202 according tothe first embodiment will be described below with reference to FIG. 6.

The moving subject region detection unit 202 calculates the movingsubject likelihood based on the interframe difference absolute valuebetween the frame of the standard image 400 and the frame of thepositioned reference image 402, corrects the moving subject likelihood,and generates the corrected moving subject likelihood. As illustrated inFIG. 6, the moving subject region detection unit 202 includes alikelihood calculation unit 600 and a likelihood correction unit 610.The likelihood correction unit 610 includes a similarity coefficientcalculation unit 611, a likelihood coefficient calculation unit 612, anintegration unit 613, and an averaging processing unit 614.

The standard image 400 and the positioned reference image 402 read fromthe above-described RAM 103 illustrated in FIG. 1 are input to thelikelihood calculation unit 600. The standard image 400 and thepositioned reference image 402 input to the likelihood calculation unit600 are images having the resolution when captured by the imagingapparatus 100 (hereinafter referred to as an resolution at a timing ofimaging) and are images not having undergone the reduction processing(resolution conversion to low resolution). Hereinafter, the resolutionat a timing of imaging is referred to as the full size. The likelihoodcalculation unit 600 calculates for each pixel the difference absolutevalue between the frame of the full-size standard image and the frame ofthe full-size positioned reference image and generates the movingsubject likelihood for each pixel based on the difference absolute valuebetween the frames for each pixel. The moving subject likelihoodgeneration processing based on the interframe difference absolute valuewill be described in detail below. The moving subject likelihood foreach pixel calculated by the likelihood calculation unit 600 istransmitted to the likelihood coefficient calculation unit 612 and theaveraging processing unit 614 of the likelihood correction unit 610.

The full-size standard image 400 read from the RAM 103 illustrated inFIG. 1 is input to the similarity coefficient calculation unit 611 ofthe likelihood correction unit 610. The similarity coefficientcalculation unit 611 detects the similarity of each pixel value of eachperipheral region to a target region including at least one pixel, i.e.,the similarity of each peripheral coordinate to the target coordinate,by using the pixel value of each pixel of the standard image 400, andacquires the similarity weight coefficient according to the similarityof these peripheral coordinates. The similarity detection processing andthe similarity weight coefficient acquisition processing will bedescribed in detail below. The similarity weight coefficient for eachperipheral coordinate obtained by the similarity coefficient calculationunit 611 is transmitted to the integration unit 613.

The likelihood coefficient calculation unit 612 acquires the movingsubject likelihood weight coefficient for each peripheral coordinatebased on the moving subject likelihood in each peripheral region for thetarget region, i.e., the moving subject likelihood of each peripheralcoordinate for the target coordinate from among the moving subjectlikelihood calculated for each pixel by the likelihood calculation unit600. The moving subject likelihood weight coefficient acquisitionprocessing will be described in detail below. The moving subjectlikelihood weight coefficient of each peripheral coordinate obtained bythe likelihood coefficient calculation unit 612 is transmitted to theintegration unit 613.

The integration unit 613 acquires the integrated weight coefficient foreach peripheral coordinate based on the similarity weight coefficientfor each peripheral coordinate obtained by the similarity coefficientcalculation unit 611 and the moving subject likelihood weightcoefficient for each peripheral coordinate obtained by the likelihoodcoefficient calculation unit 612. The integrated weight coefficientacquisition processing will be described in detail below. The integratedweight coefficient acquired by the integration unit 613 is transmittedto the averaging processing unit 614.

The averaging processing unit 614 generates the corrected moving subjectlikelihood by performing weighted addition averaging processing on themoving subject likelihoods of the target and peripheral coordinatesbased on the moving subject likelihood for each pixel obtained by thelikelihood calculation unit 600 and the integrated weight coefficientcalculated by the integration unit 613.

The moving subject likelihood generation processing and the correctedmoving subject likelihood generation processing performed by the movingsubject region detection unit 202 illustrated in FIG. 6 will bedescribed below with reference to the flowchart illustrated in FIG. 7.

Referring to FIG. 7, in step S701, the likelihood calculation unit 600of the moving subject region detection unit 202 acquires the standardimage 400 and the positioned reference image 402. Descriptions will bemade centering on the standard image 400 illustrated in FIG. 4A and thepositioned reference image 402 illustrated in FIG. 4C. As describedabove, since the standard image 400 illustrated in FIG. 4A and thepositioned reference image 402 illustrated in FIG. 4C are imagescaptured at different times, the position of a subject which has movedduring image capturing may be different between the two images. Forexample, a person 410 in the standard image 400 illustrated in FIG. 4Ahas moved to a different position of a person 420 in the positionedreference image 402 illustrated in FIG. 4C. Likewise, a person 411 inthe standard image 400 has moved to a different position of a person 421in the positioned reference image 402. The likelihood calculation unit600 compares the standard image 400 with the positioned reference image402 to detect a region where each person has moved in the images, as amoving region.

In step S702, the likelihood calculation unit 600 calculates theinterframe difference absolute value between the frame of the standardimage 400 and the frame of the positioned reference image 402 for eachpixel and obtains the moving subject likelihood for each pixel based onthe moving subject likelihood curve illustrated in FIG. 8.

FIG. 8 illustrates the moving subject likelihood curve representing therelation between the interframe difference absolute value and the movingsubject likelihood. The vertical axis denotes the moving subjectlikelihood, and the horizontal axis denotes the interframe differenceabsolute value. According to the present embodiment, a moving subjectlikelihood curve is set so that the moving subject likelihood increaseswith increasing interframe difference absolute value. With the movingsubject likelihood curve illustrated in FIG. 8, the moving subjectlikelihood is represented by a value from 0 to 200, i.e., the maximumvalue of the moving subject likelihood is 200. When the interframedifference absolute value is less than a threshold value (a thresholdvalue TH), for example, the value of the moving subject likelihoodlinearly changes between 100 and 200. When the interframe differenceabsolute value is the threshold value (the threshold value TH) or more,the value of the moving subject likelihood is fixed to 200. According tothe present embodiment, the magnitude of the value of the moving subjectlikelihood is set as illustrated in FIG. 8 due to the reason illustratedin FIG. 5. More specifically, the combination ratio curve illustrated inFIG. 5 is set so that, when the value of the corrected moving subjectlikelihood is 100, the combination ratio of the standard image 400 is100%. This means that, when the value of the corrected moving subjectlikelihood is 100 or more, the combination of the moving subject regionof the reference image 401 with the standard image 400 is inhibited. Onthe other hand, as described above, the combination ratio of thestandard image 400 is set to 50% when the value of the corrected movingsubject likelihood is 0. This setting is intended to combine astationary region of the reference image 401 with the standard image 400with a combination ratio of 50%. To perform the processing for combiningthe standard image 400 with the moving subject region of the referenceimage 401 based on the combination ratio curve illustrated in FIG. 5,the magnitudes of the values of the moving subject likelihood and thecorrected moving subject likelihood are set based on the moving subjectlikelihood curve illustrated in FIG. 8.

Referring back to the flowchart in FIG. 7, in step S703, by using thepixel value at each pixel coordinate of the standard image 400, thesimilarity coefficient calculation unit 611 detects the similarity ofeach peripheral coordinate to the target coordinate and acquires thesimilarity weight coefficient according to the similarity for eachperipheral coordinate.

The similarity detection processing and the similarity weightcoefficient acquisition processing will be described below withreference to FIGS. 9A and 9B. FIG. 9A illustrates a positionalrelationship between pixels used for the similarity detection. Eachsmall square in FIG. 9A indicates one pixel. Referring to FIG. 9A, thepixel at the black coordinate represents the pixel at the targetcoordinate, and a pixel at the white coordinate represents each pixel atthe peripheral coordinate. FIG. 9B illustrates a similarity weightcoefficient curve used when converting the similarity into thesimilarity weight coefficient. The vertical axis denotes the similarityweight coefficient and the horizontal axis denotes the similarity.

Based on the calculation represented by formula (4), the similaritycoefficient calculation unit 611 first calculates the differenceabsolute value between the pixel value at the target coordinate and thepixel value at the peripheral coordinate for each pixel and, based onthe difference absolute value for each pixel, calculates a similarity Siof each pixel at each peripheral coordinate to the pixel at the targetcoordinate illustrated in FIG. 9A. The similarity refers to anevaluation value which indicates the similarity between the pixel valueat the target coordinate and the pixel value at the peripheralcoordinate.Si=1/(|Yc−Ysi|+|Uc−Usi|+|Vc−Vsi|)  Formula (4)

Referring to formula (4), Yc denotes the luminance value of the pixel atthe target coordinate, and Uc and Vc denote the color difference valuesof the pixel at the target coordinate, Ysi denotes the luminance valueof the pixel at the peripheral coordinate, and Usi and Vsi denote thecolor difference values of the pixel at the peripheral coordinate. Morespecifically, the similarity Si decreases with increasing differenceabsolute value between the pixel value at the target coordinate and thepixel value at the peripheral coordinate and increases with decreasingdifference absolute value. When the denominator of the formula (4) is 0,i.e., when the difference absolute value between the pixel value at thetarget coordinate and the pixel value at the peripheral coordinate is 0,the similarity Si is set to 1.0.

Then, the similarity coefficient calculation unit 611 acquires thesimilarity weight coefficient based on the similarity calculated foreach peripheral coordinate and the similarity weight coefficient curveillustrated in FIG. 9B. In the example setting of the similarity weightcoefficient curve illustrated in FIG. 9B, the similarity weightcoefficient increases with increasing similarity. In the similarityweight coefficient curve illustrated in FIG. 9B, the similarity weightcoefficient is represented by a value from 0.0 to 1.0, i.e., the maximumvalue of the similarity weight coefficient is 1.0. In the example of thesimilarity weight coefficient curve illustrated in FIG. 9B, the value ofthe similarity weight coefficient is set to 0 when the similarity isless than 0.2. This setting is intended to exclude the moving subjectlikelihood of the peripheral coordinate with a low similarity to allowonly the moving subject likelihood of the peripheral coordinate with ahigh similarity to be subjected to the weighted addition averagingprocessing (described below), thus preventing the increase in the movingsubject likelihood of stationary subjects around the moving subject.

For example, in the case of the person 411 in the standard image 400illustrated in FIG. 4A, the detection result is correct if the region ofthe person 411 is detected as a moving subject region. However, when themoving subject likelihood is corrected, the moving subject likelihood oftrees and buildings, which are stationary subjects existing in theperipheral region of the person 411, may also possibly increase.Therefore, in the similarity weight coefficient curve illustrated inFIG. 9B, the similarity weight coefficient is set to 0 when thesimilarity is less than 0.2. This setting is intended to allow only themoving subject likelihood of the peripheral coordinate with a highsimilarity to be subjected to the weighted addition averagingprocessing. This makes it easier to perform moving subject likelihoodcorrection by using the moving subject likelihood of the same subjecthaving similar pixel values, preventing the increase in the movingsubject likelihood of stationary subjects around the moving subject.

Referring back to the flowchart in FIG. 7, in step S704, the likelihoodcoefficient calculation unit 612 acquires the moving subject likelihoodweight coefficient for each peripheral coordinate based on the movingsubject likelihood calculated by the likelihood calculation unit 600 andthe moving subject likelihood weight coefficient curve illustrated inFIG. 10. In the example setting of the moving subject likelihood weightcoefficient curve illustrated in FIG. 10, the moving subject likelihoodweight coefficient increases with increasing moving subject likelihood.The vertical axis denotes the moving subject likelihood weightcoefficient, and the horizontal axis denotes the moving subjectlikelihood. In the moving subject likelihood weight coefficient curveillustrated in FIG. 10, the weight coefficient is represented by a valuefrom 0.0 to 1.0, i.e., the maximum value of the moving subjectlikelihood weight coefficient is 1.0. When the moving subject likelihoodis 150 or more, the moving subject likelihood weight coefficient is setto 1.0. When the moving subject likelihood is less than 100, the movingsubject likelihood weight coefficient is set to 0.0. When the movingsubject likelihood is between 100 and 150, for example, the movingsubject likelihood weight coefficient linearly changes.

As described in step S702, when the value of the corrected movingsubject likelihood is 100 or more, the combination ratio of the standardimage becomes 100%, and the combination of the moving subject region ofthe reference image 401 is inhibited. In the example setting of themoving subject likelihood weight coefficient curve illustrated in FIG.10, the weight coefficient is larger than 0.0 when the moving subjectlikelihood is 100 or more. This setting is intended to allow only themoving subject likelihood of 100 or more of the peripheral coordinate tobe subjected to the weighted addition averaging processing so that themoving subject likelihood less than 100 is corrected to a larger value.The combination ratio of the standard image can be increased bycorrecting the moving subject likelihood less than 100 to a larger valuein this way. The moving subject likelihood larger than 100 may becorrected to a smaller value by subjecting only the moving subjectlikelihood of 100 or more of the peripheral coordinate to the weightedaddition averaging processing (described below). However, the value ofthe corrected moving subject likelihood becomes 100 or more even if themoving subject likelihood is corrected to a smaller value. Therefore,the combination ratio of the standard image 400 becomes 100%, and thecombination of the moving subject region of the reference image 401 isinhibited. More specifically, correcting the moving subject likelihoodin the way according to the present embodiment expands the region to bedetected as a moving subject region.

In step S705, the integration unit 613 acquires the integrated weightcoefficient for each peripheral coordinate based on the above-describedsimilarity weight coefficient for each peripheral coordinate and theabove-described moving subject likelihood weight coefficient for eachperipheral coordinate.

A method for acquiring the integrated weight coefficient of theperipheral coordinate will be described below with reference to FIGS.11A and 11B. FIG. 11A illustrates an integrated weight coefficient curverepresenting the relation between the product of the similarity weightcoefficient and the moving subject likelihood weight coefficient, andthe integrated weight coefficient. The vertical axis denotes theintegrated weight coefficient and the horizontal axis denotes theproduct of the two coefficients. FIG. 11B illustrates an example of thevalue of the integrated weight coefficient for each coordinatecorresponding to the target and peripheral coordinates illustrated inFIG. 9A.

The integration unit 613 calculates the product of the similarity weightcoefficient and the moving subject likelihood weight coefficient of theperipheral coordinate and, based on the integrated weight coefficientcurve illustrated in FIG. 11A, acquires the integrated weightcoefficient illustrated in FIG. 11B for each peripheral coordinate. Inthe example setting of the integrated weight coefficient curveillustrated in FIG. 11A, the integrated weight coefficient increaseswith increasing product of the similarity weight coefficient and themoving subject likelihood weight coefficient. In the integrated weightcoefficient curve illustrated in FIG. 11A, the weight coefficient isrepresented by a value from 0.0 to 1.0, i.e., the maximum value of theintegrated weight coefficient is 1.0. For example, when the similarityweight coefficient is 1.0 and the moving subject likelihood weightcoefficient is 1.0, the product of these coefficients becomes 1.0, andthe integrated weight coefficient acquired from the integrated weightcoefficient curve illustrated in FIG. 11A becomes 1.0. When either oneof the similarity weight coefficient and the moving subject likelihoodweight coefficients is 0, the product of these coefficients becomes 0,and the integrated weight coefficient becomes 0.0. More specifically,the integration unit 613 sets the integrated weight coefficient of theperipheral coordinate having a high similarity and a high moving subjectlikelihood to a large value.

By acquiring the integrated weight coefficient as described above, theintegration unit 613 can correct the moving subject likelihood of thetarget coordinate having a similar pixel value to the peripheralcoordinate and a high moving subject likelihood of the peripheralcoordinate, to a large value. This makes it easier to detect, as amoving subject region, the region of a moving subject which had beenunable to be correctly detected as a moving subject region due to thelow moving subject likelihood. In the meantime, the moving subjectlikelihood of the target coordinate not having a similar pixel value tothe peripheral coordinate will not be corrected. This allows preventingthe moving subject likelihood of a stationary region around the movingsubject from being corrected to a large value. The moving subjectlikelihood of the target coordinates having a low moving subjectlikelihood of the peripheral coordinate is not corrected. This allowspreventing the moving subject likelihood which has been changed toapproximately 30 by the influence of random noise occurring in astationary region, from being corrected to a large value.

The integrated weight coefficient acquisition processing for the targetcoordinate will be described below with reference to FIG. 11B.

The integration unit 613 sets the integrated weight coefficient of thetarget coordinate to 1.0 as the maximum value, as illustrated in FIG.11B, and sets the integrated weight coefficient of the peripheralcoordinate to the integrated weight coefficient acquired in theabove-described processing.

The integrated weight coefficient of the target coordinates is notlimited to 1.0. For example, setting the integrated weight coefficientof target coordinates to a value less than 1.0 allows performing themoving subject likelihood correction with the moving subject likelihoodof the peripheral coordinate emphasized.

Then, in step S705 illustrated in FIG. 7, the averaging processing unit614 performs the weighted addition averaging processing on the movingsubject likelihoods of the target and peripheral coordinates based onthe integrated weight coefficient to generate the corrected movingsubject likelihood. More specifically, the averaging processing unit 614performs the product-sum operation on the pixel values of the target andperipheral coordinates with the integrated weight coefficient and thendivides the product by the sum total of the integrated weightcoefficients to calculate the corrected moving subject likelihood.

The moving subject likelihood weight coefficient curve may be changedaccording to the sensitivity information during image capturing. Forexample, in the case where the sensitivity is high during imagecapturing, the amplitude of random noise of the standard image 400 andthe reference image 401 increases, and accordingly the moving subjectlikelihood tends to increase due to the random noise. For this reason,the moving subject likelihood weight coefficient may be changedaccording to the noise amount. For example, in the case where thesensitivity is high accompanied by a large noise amount, it is desirableto decrease the moving subject likelihood weight coefficient incomparison with the case of the low sensitivity during image capturing.More specifically, in the case where the sensitivity is high duringimage capturing, the averaging processing unit 614 decreases the movingsubject likelihood weight coefficient. More specifically, the averagingprocessing unit 614 decreases the slope of the moving subject likelihoodweight coefficient curve or increases the x-intercept according to thenoise amount in comparison with the case of the moving subjectlikelihood weight coefficient curve illustrated in FIG. 10.

Although, in the example according the above-described embodiment, theintegrated weight coefficient is a multivalued number from 0.0 to 1.0,the integrated weight coefficient may be a binary, for example, 0.0 and1.0. This eliminates the need of the weight coefficient multiplicationprocessing by the averaging processing unit 614 and allows reducing theamount of processing.

Although, in the example according to the above-described embodiment,one pixel is used as the pixel of the target region, the moving subjectlikelihood may be corrected by using a plurality of pixels as the targetregion. In this case, the similarity coefficient calculation unit 611calculates the similarity based on the difference absolute value betweenthe average pixel value of a plurality of pixels in the target regionand the pixel value of the peripheral region.

Although, in the example according to the above-described embodiment,the image combination based on the above-described corrected movingsubject likelihood is applied in the combination function of combining aplurality of images to reduce random noise, an example application ofthe image combination based on the corrected moving subject likelihoodis not limited thereto. For example, by combining a plurality of imagescaptured with different exposures, the above-described image combinationmay be applied to a high dynamic range (HDR) combination function forextending the dynamic range. In this case, when images after adjustingthe brightness levels of the standard image 400 and the positionedreference image 402 captured with different exposures are input to theabove-described moving subject region detection unit 202, the movingsubject region detection unit 202 calculates the corrected movingsubject likelihood and performs the image combination based on thatcorrected moving subject likelihood.

As described above, when detecting a moving subject region between aplurality of images, the image processing apparatus according to thefirst embodiment corrects the moving subject likelihood of the targetregion based on the moving subject likelihood and the similarity of theperipheral region. The present embodiment allows improving the movingsubject likelihood of a moving subject which is difficult to bedistinguished from random noise, thus improving the detection accuracyof a moving subject region.

A second embodiment will be described below.

According to the first embodiment, the moving subject region detectionunit 202 calculates the corrected moving subject likelihood based on thefull-size standard image 400 and positioned reference image 402. On theother hand, the moving subject region detection unit 202 according tothe second embodiment calculates the corrected moving subject likelihoodbased on the standard image 400 and the positioned reference image 402with a resolution different from the resolution of the full-size images.The moving subject region detection unit 202 according to the secondembodiment sets a plurality of resolutions as resolutions different fromthe resolution of the full-size images and performs hierarchicallikelihood generation processing to calculate the moving subjectlikelihood based on the standard image 400 and the positioned referenceimage 402 for each of these resolutions. Then, the moving subject regiondetection unit 202 according to the second embodiment performscorrection processing on the moving subject likelihood calculated foreach of different resolutions and, based on the corrected moving subjectlikelihood, combines a plurality of images to generate a combined image.In the following descriptions, the moving subject likelihood calculatedand corrected for each of different resolutions is referred to as ahierarchically corrected moving subject likelihood.

The moving subject region detection unit 202 according to the secondembodiment calculates the hierarchically corrected moving subjectlikelihood to prevent a stationary region around the moving subject frombeing incorrectly detected as a moving subject region and prevent randomnoise from being incorrectly detected as a moving subject region. Thisallows more accurate detection of the moving subject as a moving subjectregion.

The imaging apparatus as an example application of the image processingapparatus according to the second embodiment has a similar configurationto the above-described image processing apparatus illustrated in FIG. 1,and therefore redundant illustration and descriptions thereof will beomitted. The configuration and operations of the moving subject regiondetection unit 202 illustrated in FIG. 2 according to the secondembodiment are different from those according to the first embodiment.In the second embodiment, operations and processing identical to thosein the first embodiment are assigned the same reference numerals asthose in the first embodiment, and detailed descriptions thereof will beomitted.

FIG. 12 illustrates an example configuration of the moving subjectregion detection unit 202 according to the second embodiment. The movingsubject region detection unit 202 according to the second embodimentcalculates the interframe difference absolute value between the standardimage 400 and the positioned reference image 402 for each of differentresolutions and, based on the interframe difference absolute valuecalculated for each resolution, calculates the hierarchically correctedmoving subject likelihood. Therefore, the moving subject regiondetection unit 202 according to the second embodiment includeslikelihood calculation units 1200 to 1202, edge degree calculation units1210 and 1211, the likelihood correction unit 610, enlargementprocessing units 1220 to 1224, combination ratio calculation units 1230and 1231, and likelihood combination units 1240 and 1241. The likelihoodcorrection unit 610 illustrated in FIG. 12 has a similar configurationto the above-described likelihood correction unit 610 illustrated inFIG. 6.

The full-size standard image 400 and positioned reference image 402similar to those according to the first embodiment are read from the RAM103 illustrated in FIG. 1 and then input to the likelihood calculationunit 1200. Similarly to the likelihood calculation unit 600 illustratedin FIG. 6, the likelihood calculation unit 1200 calculates thedifference absolute value between the frame of the standard image 400and the frame of the positioned reference image 402 for each pixel, andcalculates the moving subject likelihood for each pixel based on thedifference absolute value between the frames for each pixel. Accordingto the present embodiment, the moving subject likelihood obtained by thelikelihood calculation unit 1200 is referred to as a full-size movingsubject likelihood. The full-size moving subject likelihood output fromthe likelihood calculation unit 1200 is transmitted to the likelihoodcombination unit 1240.

A ¼-size standard image and a ¼-size positioned reference image havingthe ¼ resolution converted from the standard image 400 and thepositioned reference image 402 having the resolution at a timing ofimaging, respectively, are input to the likelihood calculation unit1201. For example, processing for converting the standard image 400 andthe positioned reference image 402 having the resolution at a timing ofimaging into the ¼ resolution is performed by a resolution converter(not illustrated), and these ¼-size images are stored in the RAM 103.The resolution converter (not illustrated) may be provided, for example,in the moving subject region detection unit 202 or formed in the controlunit 101 illustrated in FIG. 1. The likelihood calculation unit 1201calculates the difference absolute value between the frames of the¼-size standard image 400 and positioned reference image 402 for eachpixel, and calculates the moving subject likelihood for each pixel basedon the difference absolute value for each pixel. According to thepresent embodiment, the hierarchical moving subject likelihood obtainedfor a hierarchy having the ¼-size resolution by the likelihoodcalculation unit 1201 is referred to as a ¼-size moving subjectlikelihood. The ¼-size moving subject likelihood output from thelikelihood calculation unit 1201 is transmitted to the likelihoodcorrection unit 610.

A 1/16-size standard image and a 1/16-size positioned reference imagehaving the 1/16 resolution converted from the standard image 400 and thepositioned reference image 402 having the resolution at a timing ofimaging, respectively, are input to the likelihood calculation unit1202. Like the above-described processing, for example, processing forconverting the standard image 400 and the positioned reference image 402having the resolution at a timing of imaging into the 1/16 resolution isperformed by a resolution converter (not illustrated), and these1/16-size images are stored in the RAM 103. The likelihood calculationunit 1202 calculates the interframe difference absolute value betweenthe 1/16-size standard image and positioned reference image andcalculates the moving subject likelihood for each pixel based on thedifference absolute value. According to the present embodiment, thehierarchical moving subject likelihood obtained for a hierarchy havingthe 1/16-size resolution by the likelihood calculation unit 1202 isreferred to as a 1/16-size moving subject likelihood. The 1/16-sizemoving subject likelihood output from the likelihood calculation unit1202 is transmitted to the enlargement processing unit 1223.

The 1/16-size standard image is enlarged into an image equivalent to the¼ size by the enlargement processing unit 1224, and the enlarged imageis transmitted to the likelihood correction unit 610.

The likelihood correction unit 610 performs similar processing to theprocessing according to the first embodiment based on the¼-size-equivalent standard image enlarged from the 1/16-size standardimage by the enlargement processing unit 1224 and the ¼-size movingsubject likelihood from the likelihood calculation unit 1201. Thecorrected moving subject likelihood (¼-size corrected moving subjectlikelihood) generated by the likelihood correction unit 610 istransmitted to the likelihood combination unit 1241.

The edge degree calculation unit 1210 calculates the edge intensity foreach pixel for the ¼-size standard image and positioned reference imageand, based on each edge intensity calculated, acquires the edge degree(hereinafter referred to as a ¼-size edge degree) for each pixel.Likewise, the edge degree calculation unit 1211 calculates the edgeintensity for each pixel for the 1/16-size standard image and positionedreference image and, based on each edge intensity calculated, acquiresthe edge degree (hereinafter referred to as a 1/16-size edge degree) foreach pixel. The edge intensity calculation processing and the edgedegree acquisition processing according to the edge intensity will bedescribed in detail below. The ¼-size edge degree obtained by the edgedegree calculation unit 1210 is transmitted to the enlargementprocessing unit 1220, and the 1/16-size edge degree obtained by the edgedegree calculation unit 1211 is transmitted to the enlargementprocessing unit 1222.

The enlargement processing unit 1220 enlarges the ¼-size edge degreeinto the edge degree equivalent to the full size and transmits theenlarged edge degree to the combination ratio calculation unit 1230.Meanwhile, the enlargement processing unit 1222 enlarges the 1/16-sizeedge degree into the edge degree equivalent to the ¼ size and transmitsthe enlarged edge degree to the combination ratio calculation unit 1231.Meanwhile, the enlargement processing unit 1223 enlarges the 1/16-sizemoving subject likelihood into the moving subject likelihood equivalentto the ¼ size and transmits the enlarged moving subject likelihood tothe likelihood combination unit 1241. Examples of enlargement processingmethods performed by the above-described enlargement processing unitsinclude the bilinear enlargement method and the bicubic enlargementmethod.

The combination ratio calculation unit 1230 sets the combination ratioof the corrected moving subject likelihood required for the likelihoodcombination unit 1240 (described below) to combine a corrected movingsubject likelihood, based on the full-size edge degree enlarged from the¼-size edge degree. Meanwhile, the combination ratio calculation unit1231 sets the combination ratio of the corrected moving subjectlikelihood required for the likelihood combination unit 1241 (describedbelow) to combine a corrected moving subject likelihood, based on the¼-size edge degree enlarged from the 1/16-size edge degree. Thecombination ratio of the corrected moving subject likelihood acquired bythe combination ratio calculation unit 1230 is transmitted to thelikelihood combination unit 1240, and the combination ratio of thecorrected moving subject likelihood acquired by the combination ratiocalculation unit 1231 is transmitted to the likelihood combination unit1241.

The likelihood combination unit 1241 combines the moving subjectlikelihood from the enlargement processing unit 1223 or the correctedmoving subject likelihood from the likelihood correction unit 610 basedon the combination ratio of the corrected moving subject likelihood fromthe combination ratio calculation unit 1231. Then, the corrected movingsubject likelihood resulting from the combination processing by thelikelihood combination unit 1241 is transmitted to the enlargementprocessing unit 1221 as a ¼-size hierarchically corrected moving subjectlikelihood. Like the enlargement processing units, the enlargementprocessing unit 1221 enlarges the ¼-size hierarchically corrected movingsubject likelihood into a size equivalent to the full size. Then, thehierarchically corrected moving subject likelihood resulting from theenlargement processing by the enlargement processing unit 1221 istransmitted to the likelihood combination unit 1240.

The likelihood combination unit 1240 combines the full-size movingsubject likelihood from the likelihood calculation unit 1200 or thecorrected moving subject likelihood from the enlargement processing unit1221 based on the combination ratio of the corrected moving subjectlikelihood from the combination ratio calculation unit 1230. Then, thecorrected moving subject likelihood resulting from the combinationprocessing by the likelihood combination unit 1240 is transmitted to theimage combination unit 203 illustrated in FIG. 2 according to the secondembodiment as the full-size hierarchically corrected moving subjectlikelihood.

Like the first embodiment, the image combination unit 203 according tothe second embodiment sets the combination ratio based on the full-sizehierarchically corrected moving subject likelihood and, based on thecombination ratio, combines the full-size standard image and positionedreference image for each pixel to generate a combined image.

The moving subject likelihood generation processing and the correctedmoving subject likelihood generation processing performed by the movingsubject region detection unit 202 according to the second embodimentwill be described below with reference to the flowchart illustrated inFIG. 13.

Referring to FIG. 13, in step S1301, the moving subject region detectionunit 202 acquires a full-size standard image (for example, the standardimage 400 illustrated in FIG. 4A) and a full-size positioned referenceimage (for example, the positioned reference image 402 illustrated inFIG. 4C).

In step S1302, for example, a resolution converter (not illustrated)performs the reduction processing using smoothing processing and pixelthinning processing on the full-size standard image and positionedreference image to generate the above-described ¼-size and 1/16-sizelow-resolution images.

The reduction processing will be described below with reference to FIGS.14, 15A, and 15B.

FIG. 14 illustrates an example pixel value of a full-size standard image1600 and an example pixel value of a full-size positioned referenceimage 1601. Referring to FIG. 14, the vertical axis denotes the pixelvalue and the horizontal axis denotes the horizontal coordinate. Thedotted line 1602 denotes an example pixel value of the full-sizestandard image 1600, and the solid line 1603 denotes an example pixelvalue of the full-size positioned reference image 1601.

According to the present embodiment, the reduction processing isperformed on the full-size images illustrated in FIG. 14 to generate a¼-size image and a 1/16-size image having horizontal and verticalresolutions ¼ and 1/16 times the resolution of the full-size images,respectively. Examples of reduction methods include the bilinearreduction method and the bicubic reduction method.

Referring to FIGS. 15A and 15B, the vertical axis denotes the pixelvalue and the horizontal axis denotes the horizontal coordinate. FIG.15A illustrates a 1/16-size image, and FIG. 15B illustrates a ¼-sizeimage. Referring to FIG. 15A, the dotted line 1700 denotes an examplepixel value of the 1/16-size standard image, and the solid line 1701denotes an example pixel value of the 1/16-size positioned referenceimage. Referring to FIG. 15B, the dotted line 1702 denotes an examplepixel value of the ¼-size standard image, and the solid line 1703denotes an example pixel value of the ¼-size positioned reference image.The pixel value (the dotted line 1700) of the 1/16-size standard imageillustrated in FIG. 15A is smoothed to a further extent than the pixelvalue (the dotted line 1702) of the ¼-size standard image illustrated inFIG. 15B through the reduction processing.

Referring back to the flowchart illustrated in FIG. 13, in step S1303,each of the likelihood calculation units 1200, 1201, and 1202 obtainsthe interframe difference absolute value between the standard image 400and the positioned reference image 402 having the correspondingresolutions and calculates the moving subject likelihood based on thedifference absolute value.

More specifically, the likelihood calculation unit 1200 calculates theinterframe difference absolute value for each pixel between thefull-size standard image and the full-size positioned reference imageand calculates the full-size moving subject likelihood based on themoving subject likelihood curve illustrated in FIG. 8.

The likelihood calculation unit 1201 also calculates the interframedifference absolute value for each pixel between the ¼-size standardimage and positioned reference image and acquires the ¼-size movingsubject likelihood based on the moving subject likelihood curveillustrated in FIG. 8. FIG. 15D illustrates an example of the ¼-sizemoving subject likelihood. The vertical axis denotes the moving subjectlikelihood and the horizontal axis denotes the horizontal coordinate.The ¼-size moving subject likelihood illustrated in FIG. 15D is themoving subject likelihood acquired based on the ¼-size image illustratedin FIG. 15B.

Likewise, the likelihood calculation unit 1202 calculates the interframedifference absolute value for each pixel between the 1/16-size standardimage and positioned reference image and acquires the 1/16-size movingsubject likelihood based on the moving subject likelihood curveillustrated in FIG. 8. FIG. 15C illustrates an example of the 1/16-sizemoving subject likelihood. The vertical axis denotes the moving subjectlikelihood and the horizontal axis denotes the horizontal coordinate.The 1/16-size moving subject likelihood illustrated in FIG. 15C is themoving subject likelihood acquired based on the 1/16-size imageillustrated in FIG. 15A.

In step S1304 in FIG. 13, the likelihood correction unit 610 correctsthe ¼-size moving subject likelihood based on the ¼-size-equivalentstandard image horizontally and vertically enlarged from the 1/16-sizestandard image four times by the enlargement processing unit 1224, togenerate the ¼-size corrected moving subject likelihood. FIG. 15Fillustrates an example of the ¼-size corrected moving subjectlikelihood. The vertical axis denotes the corrected moving subjectlikelihood and the horizontal axis denotes the horizontal coordinate.The ¼-size corrected moving subject likelihood illustrated in FIG. 15Fis the corrected moving subject likelihood which is obtained bycorrecting the ¼-size moving subject likelihood illustrated in FIG. 15Dbased on the ¼-size-equivalent standard image enlarged from the1/16-size standard image illustrated in FIG. 15A by the enlargementprocessing unit 1224. A method for correcting the ¼-size moving subjectlikelihood by the likelihood correction unit 610 is similar to theprocessing according to the first embodiment, and therefore redundantdescriptions thereof will be omitted.

In step S1305 in FIG. 13, each of the edge degree calculation units 1210and 1211 acquires the edge degree based on the edge intensity calculatedfrom the standard image 400 and the positioned reference image 402having the corresponding resolution.

Edge intensity calculation processing will be described below. The edgeintensity can be calculated, for example, by the Sobel filterprocessing. The edge intensity calculation processing based on the Sobelfilter processing will be described below.

In the edge intensity calculation processing, each of the edge degreecalculation units 1210 and 1211 first multiplies the pixel value of eachof the nine coordinates around the target coordinate by the coefficientrepresented by formula (5) and then totals the multiplication results tocalculate a vertical edge intensity Sv. Likewise, each of the edgedegree calculation units 1210 and 1211 multiplies the pixel value ofeach of the nine coordinates around the target coordinate by thecoefficient represented by formula (6) and then totals themultiplication results to calculate a horizontal edge intensity Sh.

$\begin{matrix}{{Sv} = \begin{pmatrix}{- 1} & {- 2} & {- 1} \\0 & 0 & 0 \\1 & 2 & 1\end{pmatrix}} & {{Formula}\mspace{14mu}(5)} \\{{Sh} = \begin{pmatrix}{- 1} & 0 & 1 \\{- 2} & 0 & 2 \\{- 1} & 0 & 1\end{pmatrix}} & {{Formula}\mspace{14mu}(6)}\end{matrix}$

Then, in the edge intensity calculation processing, each of the edgedegree calculation units 1210 and 1211 calculates an edge intensity Ewhich equals the square root of the sum of the squares of the verticaledge intensity Sv and the horizontal edge intensity Sh, represented byformula (7).E=√{square root over (Sh ² +Sv ²)}  Formula (7)

A method for acquiring the edge degree based on the edge intensitycalculated as described above will be described below with reference toFIG. 16. FIG. 16 illustrates the edge degree curve used for obtainingthe edge degree from the edge intensity. The vertical axis denotes theedge degree and the horizontal axis denotes the edge intensity.

The edge degree calculation unit 1210 calculates the edge intensity foreach pixel as described above for the ¼-size standard image andpositioned reference image and selects the higher edge intensity foreach pixel. Then, based on the edge degree curve illustrated in FIG. 16,the edge degree calculation unit 1210 acquires the edge degree accordingto the edge intensity selected for each pixel. In this way, the edgedegree obtained by the edge degree calculation unit 1210 becomes the¼-size edge degree.

Likewise, the edge degree calculation unit 1211 calculates the edgeintensity for each pixel for the 1/16-size standard image and positionedreference image and selects the higher edge intensity for each pixel.Then, based on the edge degree curve illustrated in FIG. 16, the edgedegree calculation unit 1211 acquires the edge degree according to theedge intensity selected for each pixel. In this way, the edge degreeobtained by the edge degree calculation unit 1211 becomes the 1/16-sizeedge degree. FIG. 15E illustrates an example of the 1/16-size edgedegree. The vertical axis denotes the edge degree and the horizontalaxis denotes the horizontal coordinate. The 1/16-size edge degreeillustrated in FIG. 15E is the edge degree obtained based on the1/16-size image illustrated in FIG. 15A through the edge degreeacquisition processing described in step S1305.

In step S1306 in FIG. 13, each of the combination ratio calculation unit1230 and 1231 obtains the combination ratio of the corrected movingsubject likelihood required for the likelihood combination units 1240and 1241 to combine the corrected moving subject likelihood,respectively, based on the edge degree acquired as described above.

In step S1307, each of the likelihood combination unit 1240 and 1241combines the corrected moving subject likelihood or the moving subjectlikelihood based on the combination ratio of the corrected movingsubject likelihood acquired in step S1306, to generate thehierarchically corrected moving subject likelihood.

In step S1308, the likelihood combination unit 1240 determines whetherthe combination of the corrected moving subject likelihood is completedfor all sizes (all resolutions). When the likelihood combination unit1240 determines that the combination of the corrected moving subjectlikelihood is not completed for all sizes (NO in step S1308), theprocessing returns to step S1306. The likelihood combination unit 1240repeats the processing in steps S1306 and S1307 until it determines thatthe combination of the corrected moving subject likelihood is completedfor all sizes.

Moving subject likelihood combination processing for each resolution(each size) in steps S1306, S1307, and S1308 illustrated in FIG. 13 willbe described below.

First of all, ¼-size corrected moving subject likelihood combinationprocessing will be described below with reference to FIGS. 12, 15G, 15H,and 17.

The combination ratio calculation unit 1231 sets the combination ratioof the ¼-size corrected moving subject likelihood to be combined by thelikelihood combination unit 1241 based on the ¼-size edge degreehorizontally and vertically enlarged from the 1/16-size edge degree fourtimes by the enlargement processing unit 1222. When the 1/16-size edgedegree and moving subject likelihood are horizontally and verticallyenlarged four times into the ¼-size-equivalent resolution as describedabove, the combination ratio of the corrected moving subject likelihoodacquired by the combination ratio calculation unit 1231 also becomes the¼-size-equivalent resolution.

FIG. 15G illustrates an example of the combination ratio of the ¼-sizecorrected moving subject likelihood. The vertical axis denotes thecombination ratio of the ¼-size corrected moving subject likelihood, andthe horizontal axis denotes the horizontal coordinate. The combinationratio of the ¼-size corrected moving subject likelihood illustrated inFIG. 15G is the combination ratio acquired from the curve illustrated inFIG. 17 based on the ¼-size-equivalent edge degree enlarged from the1/16-size edge degree illustrated in FIG. 15E.

Then, based on the combination ratio of the ¼-size corrected movingsubject likelihood, the likelihood combination unit 1241 combines the¼-size moving subject likelihood and the ¼-size-equivalent movingsubject likelihood enlarged from the 1/16-size moving subject likelihoodby the enlargement processing unit 1223, by using formula (8). In thisway, the ¼-size hierarchically corrected moving subject likelihood iscalculated.MC4=wm4*M4+(1−wm4)*M16  Formula (8)

Referring to the formula (8), M4 denotes the ¼-size corrected movingsubject likelihood, M16 denotes the moving subject likelihoodhorizontally and vertically enlarged from the 1/16-size moving subjectlikelihood four times by the enlargement processing unit 1223, wm4denotes the combination ratio of the ¼-size corrected moving subjectlikelihood, and MC4 denotes the ¼-size hierarchically corrected movingsubject likelihood.

FIG. 15H illustrates an example of the ¼-size hierarchically correctedmoving subject likelihood. The vertical axis denotes the ¼-sizehierarchically corrected moving subject likelihood, and the horizontalaxis denotes the horizontal coordinate. The ¼-size hierarchicallycorrected moving subject likelihood illustrated in FIG. 15H is obtainedby combining the ¼-size corrected moving subject likelihood illustratedin FIG. 15F and the 1/16-size moving subject likelihood illustrated inFIG. 15C based on the combination ratio of the ¼-size corrected movingsubject likelihood illustrated in FIG. 15G.

FIG. 17 illustrates an example of the combination ratio curve of the¼-size hierarchically corrected moving subject likelihood. The verticalaxis denotes the combination ratio and the horizontal axis denotes theedge degree. As illustrated in FIG. 17, the combination ratio curve ofthe ¼-size hierarchically corrected moving subject likelihood is set sothat the combination ratio of the ¼-size hierarchically corrected movingsubject likelihood increases with increasing edge degree. With the¼-size hierarchically corrected moving subject likelihood, thecombination ratio of the 1/16-size moving subject likelihood increasesin flat portions having a low edge degree, and the combination ratio ofthe ¼-size corrected moving subject likelihood increases in edgeportions having a high edge degree.

In addition, random noise in a 1/16-size image is reduced to a lowerlevel than random noise in a ¼-size image through the smoothingprocessing in the reduction processing. Therefore, with the 1/16-sizemoving subject likelihood, random noise is less likely to be detected asa motion than with the ¼-size moving subject likelihood. Morespecifically, in flat portions having a low edge degree, the ¼-sizehierarchically corrected moving subject likelihood not easily affectedby random noise can be generated by increasing the combination ratio ofthe 1/16-size moving subject likelihood not easily affected by randomnoise.

More specifically, the combination ratio of the ¼-size corrected movingsubject likelihood illustrated in FIG. 15G is 0% in a region where the1/16-size edge degree illustrated in FIG. 15E is low. Therefore, in flatportions having a low edge degree, the combination ratio of the1/16-size moving subject likelihood is 100% which suppresses theinfluence of random noise.

On the other hand, the contour of a moving subject (the boundary betweenthe moving subject and the background) has a high edge degree andtherefore is applied with a large combination ratio of the ¼-sizecorrected moving subject likelihood. This allows generating thehierarchically corrected moving subject likelihood more closelyassociated with the contour of the moving subject than using the1/16-size moving subject likelihood after being horizontally andvertically enlarged four times.

More specifically, the ¼-size hierarchically corrected moving subjectlikelihood illustrated in FIG. 15H is sharper than the 1/16-size movingsubject likelihood illustrated in FIG. 15C. This is because thecombination ratio of the ¼-size corrected moving subject likelihoodillustrated in FIG. 15G has increased under the influence of theincreased 1/16-size edge degree illustrated in FIG. 15E in the vicinityof the contour of the moving subject.

The full-size moving subject likelihood combination processing isperformed in a similar way to the ¼-size moving subject likelihoodcombination processing. More specifically, based on the full-size edgedegree horizontally and vertically enlarged from the ¼-size edge degreefour times, the likelihood combination unit 1240 combines the full-sizemoving subject likelihood and the hierarchically corrected movingsubject likelihood horizontally and vertically enlarged from the ¼-sizehierarchically corrected moving subject likelihood four times togenerate the full-size hierarchically corrected moving subjectlikelihood.

Although the second embodiment has been described above centering on anexample where only the ¼-size moving subject likelihood, i.e., themoving subject likelihood calculated from low-resolution images iscorrected, the processing is not limited thereto. For example, it isalso possible to correct the 1/16-size moving subject likelihoodcorresponding to images having a resolution lower than the ¼ size. Likethe first embodiment, the moving subject likelihood corresponding tofull-size images may be corrected also in the second embodiment.

For example, the full-size moving subject likelihood not havingundergone the reduction processing has a tendency that random noise hasa large amplitude and an increased moving subject likelihood. Therefore,when correcting the full-size moving subject likelihood, it is desirablethat the moving subject likelihood weight coefficient acquired by thelikelihood coefficient calculation unit 612 is made smaller than the ¼size. More specifically, the moving subject likelihood weightcoefficient is made smaller than the ¼ size by decreasing the slope ofthe moving subject likelihood weight coefficient curve illustrated inFIG. 10 or increasing the x-intercept.

Since the 1/16-size moving subject likelihood has a resolution lowerthan the full size, the result of correcting the 1/16-size movingsubject likelihood affects a wide region when converted into thefull-size. Therefore, if the similarity between the moving subject andthe background is high, the peripheral region of the moving subjectregion may also possibly be detected as a moving subject region.Therefore, when correcting the 1/16-size moving subject likelihood, theaveraging processing unit 614 may perform the weighted additionaveraging processing on the moving subject likelihood of a smallerperipheral region at the time of the weighted addition averagingprocessing on the moving subject likelihood of the peripheral region.For example, when correcting the ¼-size moving subject likelihood, theaveraging processing unit 614 performs the weighted addition averagingprocessing on the moving subject likelihood of the peripheral regioncomposed of 5×5 pixels. When correcting the 1/16-size moving subjectlikelihood, the averaging processing unit 614 performs the weightedaddition averaging processing on the moving subject likelihood of theperipheral region composed of 3×3 pixels.

As described above, the imaging apparatus 100 according to the secondembodiment uses the hierarchically corrected moving subject likelihoodto prevent a stationary region around the moving subject from beingincorrectly detected as a moving subject region and prevent random noisefrom being incorrectly detected as a moving subject region. Therefore,the imaging apparatus 100 according to the second embodiment makes itpossible to more accurately detect a moving subject as a moving subjectregion.

The above-described imaging apparatus 100 according to the first and thesecond embodiments is applicable to digital cameras, digital camcorders,portable terminals (such as smart phones and tablet terminals havingcamera functions), monitoring cameras, industrial cameras, onboardcameras, and medical camera.

The processing by the image processing unit according to theabove-described embodiments may be performed by hardware configurations,or a part of the processing may be implemented by softwareconfigurations and the remaining part thereof may be implemented byhardware configurations. When the processing is performed by software,for example, the processing is implemented when the CPU executes aprogram stored in the ROM.

The present disclosure can also be achieved when a program forimplementing at least one of the functions according to theabove-described embodiments is supplied to a system or apparatus via anetwork or storage medium, and at least one processor in a computer ofthe system or apparatus reads and executes the program. Further, thepresent disclosure can also be achieved by a circuit (for example, anapplication specific integrated circuit (ASIC)) for implementing atleast one function.

The above-described embodiments are to be considered as illustrative inembodying the present disclosure, and not restrictive of the technicalscope of the present disclosure. The present disclosure may be embodiedin diverse forms without departing from the technical concepts oressential characteristics thereof.

While the present disclosure has been described with reference toembodiments, it is to be understood that the disclosure is not limitedto the disclosed embodiments. The scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

OTHER EMBODIMENTS

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may include one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

This application claims the benefit of Japanese Patent Application No.2018-027878, filed Feb. 20, 2018, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus to detect a movingsubject region, the image processing apparatus comprising: a memory thatstores instructions; and one or more processors configured to executethe instructions to cause the image processing apparatus to function as:a likelihood generation unit configured to detect a motion of a regionin a first image and, based on at least two input images, generate amoving subject likelihood for each region in the first image, asimilarity detection unit configured to detect, as a detectedsimilarity, a similarity between a target region and a peripheral regionof the target region for at least one of the at least two input images,and a correction unit configured to correct the generated moving subjectlikelihood of the target region based on the detected similarity and thegenerated moving subject likelihood of the peripheral region, whereinthe moving subject region is detected based on the corrected movingsubject likelihood of the target region.
 2. The image processingapparatus according to claim 1, wherein the correction unit corrects thegenerated moving subject likelihood of the target region to increase themoving subject likelihood of the target region.
 3. The image processingapparatus according to claim 1, wherein the likelihood generation unitis configured to use at least one of the at least two input images as astandard image for detecting the motion of the region in the firstimage, and wherein the similarity detection unit is configured to detecta similarity between a target region and a peripheral region of thetarget region for the standard image.
 4. The image processing apparatusaccording to claim 1, wherein executing the instructions further causesthe image processing apparatus to function as an image combination unitconfigured to combine the at least two input images based on thecorrected moving subject likelihood of the target region.
 5. The imageprocessing apparatus according to claim 1, wherein the likelihoodgeneration unit is configured to generate a moving subject likelihood ofat least one pixel of the target region and of the peripheral region,wherein the similarity detection unit is configured to detect, as apixel value similarity, a similarity between a pixel value of the atleast one pixel of the target region and a pixel value of the at leastone pixel of the peripheral region, and wherein the correction unitcorrects the generated moving subject likelihood of the at least onepixel of the target region based on the pixel value similarity and themoving subject likelihood of the at least one pixel of the peripheralregion.
 6. The image processing apparatus according to claim 5, whereinthe similarity detection unit is configured to calculate the pixel valuesimilarity based on a difference between the pixel value of the at leastone pixel of the target region and the pixel value of the at least onepixel of the peripheral region.
 7. The image processing apparatusaccording to claim 5, wherein the similarity detection unit isconfigured to calculate the pixel value similarity based on a differencebetween an average pixel value of the at least one pixel of the targetregion and the pixel value of the at least one pixel of the peripheralregion.
 8. The image processing apparatus according to claim 1, whereinexecuting the instructions further causes the image processing apparatusto function as a coefficient acquisition unit configured to acquire aweight coefficient for correcting the generated moving subject,likelihood of the target region, and wherein the correction unitcorrects the generated moving subject likelihood of the target regionbased on the acquired weight coefficient.
 9. The image processingapparatus according to claim 8, wherein the coefficient acquisition unitis configured to calculate a weight coefficient for the target andperipheral regions, and wherein, based on the calculated weightcoefficient, the correction unit performs weighted addition averagingprocessing on the generated moving subject likelihoods of the target andperipheral regions to correct the generated moving subject likelihood ofthe target region.
 10. The image processing apparatus according to claim8, wherein the coefficient acquisition unit is configured to calculate aweight coefficient based on the detected similarity and the generatedmoving subject likelihood of the peripheral region.
 11. The imageprocessing apparatus according to claim 8, wherein the coefficientacquisition unit is configured to increase an acquired weightcoefficient of the generated moving subject likelihood of the peripheralregion with an increase in the generated moving subject likelihood ofthe peripheral region.
 12. The image processing apparatus according toclaim 8, wherein the coefficient acquisition unit is configured toincrease an acquired weight coefficient of the generated moving subjectlikelihood of the peripheral region with an increase in the similaritybetween the target region and the peripheral region.
 13. The imageprocessing apparatus according to claim 8, wherein, in a case where thegenerated moving subject likelihood of the peripheral region is high andthe similarity of the peripheral region to the target region is high,the coefficient acquisition unit increases an acquired weightcoefficient of the generated moving subject likelihood of the peripheralregion.
 14. The image processing apparatus according to claim 8, whereinthe coefficient acquisition unit is configured to change the acquiredweight coefficient according to a noise amount contained in at least oneof the at least two input images.
 15. The image processing apparatusaccording to claim 1, wherein the target region is one pixel.
 16. Theimage processing apparatus according to claim 1, wherein executing theinstructions further causes the image processing apparatus to functionas: a hierarchical likelihood generation unit configured to generate amoving subject likelihood based on a low-resolution image obtained byconverting at least one of the at least two input images into a lowresolution, an edge degree acquisition unit configured to acquire anedge degree based on at least one of an edge intensity of the at leastone of the at least two input images and an edge intensity of thelow-resolution image, a ratio setting unit configured to set acombination ratio based on the edge degree, and a likelihood combinationunit configured to combine, according to the set combination ratio, themoving subject likelihood generated by the likelihood generation unitand the moving subject likelihood generated by the hierarchicallikelihood generation unit.
 17. The image processing apparatus accordingto claim 16, wherein the hierarchical likelihood generation unit isconfigured to generate a plurality of the low-resolution imagesconverted to provide a plurality of different resolutions.
 18. The imageprocessing apparatus according to claim 16, wherein the correction unitis configured to correct the moving subject likelihood generated by thehierarchical likelihood generation unit.
 19. A method for an imageprocessing apparatus to detect a moving subject region, the methodcomprising: detecting a motion of a region in a first image and, basedon at least two input images, generating a moving subject likelihood foreach region in the first image; detecting, as a detected similarity, asimilarity between a target region and a peripheral region of the targetregion for at least one of the at least two input images; and correctingthe generated moving subject likelihood of the target region based onthe detected similarity and the generated moving subject likelihood ofthe peripheral region, wherein the moving subject region is detectedbased on the corrected moving subject likelihood of the target region.20. A non-transitory storage medium storing a program to cause acomputer to perform a method for an image processing apparatus to detecta moving subject region, the method comprising: detecting a motion of aregion in a first image and, based on at least two input images,generating a moving subject likelihood for each region in the firstimage; detecting, as a detected similarity, a similarity between atarget region and a peripheral region of the target region for at leastone of the at least two input images; and correcting the generatedmoving subject likelihood of the target region based on the detectedsimilarity and the generated moving subject likelihood of the peripheralregion, wherein the moving subject region is detected based on thecorrected moving subject likelihood of the target region.